

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  <meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />

  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>Analyzing and Debugging A Teuthology Job &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../../../_static/css/custom.css" type="text/css" />

  
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../../../" src="../../../../_static/documentation_options.js"></script>
        <script src="../../../../_static/jquery.js"></script>
        <script src="../../../../_static/_sphinx_javascript_frameworks_compat.js"></script>
        <script data-url_root="../../../../" id="documentation_options" src="../../../../_static/documentation_options.js"></script>
        <script src="../../../../_static/doctools.js"></script>
        <script src="../../../../_static/sphinx_highlight.js"></script>
    
    <script type="text/javascript" src="../../../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../../../genindex/" />
    <link rel="search" title="Search" href="../../../../search/" />
    <link rel="next" title="Integration Tests for Kernel Development" href="../tests-integration-testing-teuthology-kernel/" />
    <link rel="prev" title="Integration Tests using Teuthology Workflow" href="../tests-integration-testing-teuthology-workflow/" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    <div role="navigation" aria-label="Page navigation">
  <ul class="wy-breadcrumbs">
      <li><a href="../../../../" class="icon icon-home" aria-label="Home"></a></li>
          <li class="breadcrumb-item"><a href="../../">向 Ceph 贡献：开发者指南</a></li>
          <li class="breadcrumb-item"><a href="../">Teuthology 用户指南</a></li>
      <li class="breadcrumb-item active">Analyzing and Debugging A Teuthology Job</li>
      <li class="wy-breadcrumbs-aside">
            <a href="../../../../_sources/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-debugging-tips.rst.txt" rel="nofollow"> View page source</a>
      </li>
  </ul>
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../../../" class="icon icon-home"> Ceph
          

          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../../../start/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../install/">安装 Ceph</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../cephadm/">Cephadm</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../rados/">Ceph 存储集群</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../cephfs/">Ceph 文件系统</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../monitoring/">监控概览</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../architecture/">体系结构</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../../">开发者指南</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../../intro/">简介</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../essentials/">必备知识</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../merging/">何时、合并了什么</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../issue-tracker/">问题追踪器</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../basic-workflow/">基本工作流</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../tests-unit-tests/">测试：单元测试</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../">测试：集成测试(Teuthology)</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="../tests-integration-testing-teuthology-intro/">Introduction</a></li>
<li class="toctree-l3"><a class="reference internal" href="../tests-integration-testing-teuthology-workflow/">Workflow</a></li>
<li class="toctree-l3 current"><a class="current reference internal" href="#">Debugging Tips</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#viewing-test-results">Viewing Test Results</a></li>
<li class="toctree-l4"><a class="reference internal" href="#identifying-failed-jobs">Identifying Failed Jobs</a></li>
<li class="toctree-l4"><a class="reference internal" href="#triaging-the-cause-of-failure">Triaging the cause of failure</a></li>
<li class="toctree-l4"><a class="reference internal" href="#reporting-the-issue">Reporting the Issue</a></li>
<li class="toctree-l4"><a class="reference internal" href="#debugging-an-issue-using-interactive-on-error">Debugging an issue using interactive-on-error</a></li>
<li class="toctree-l4"><a class="reference internal" href="#suggested-resources">Suggested Resources</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../tests-integration-testing-teuthology-kernel/">Kernel Development</a></li>
<li class="toctree-l3"><a class="reference internal" href="../tests-sentry-developers-guide/">Sentry Notes</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../running-tests-locally/">测试：在本地运行测试</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../tests-windows/">测试: Windows</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../dash-devel/">Ceph Dashboard 开发者文档 (之前是 HACKING.rst)</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../jaegertracing/">Tracing 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../../cephadm/">Cephadm 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../debugging-gdb/">用 GDB 调试</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../../internals/">Ceph 内幕</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../hardware-monitoring/">硬件监控</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <section id="analyzing-and-debugging-a-teuthology-job">
<span id="tests-integration-testing-teuthology-debugging-tips"></span><h1>Analyzing and Debugging A Teuthology Job<a class="headerlink" href="#analyzing-and-debugging-a-teuthology-job" title="Permalink to this heading"></a></h1>
<p>To learn more about how to schedule an integration test, refer to <a class="reference external" href="../tests-integration-testing-teuthology-workflow/#scheduling-test-run">Scheduling
Test Run</a>.</p>
<section id="viewing-test-results">
<h2>Viewing Test Results<a class="headerlink" href="#viewing-test-results" title="Permalink to this heading"></a></h2>
<p>When a teuthology run has been completed successfully, use <a class="reference external" href="https://pulpito.ceph.com">pulpito</a> dasboard
to view the results:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">pulpito</span><span class="o">.</span><span class="n">front</span><span class="o">.</span><span class="n">sepia</span><span class="o">.</span><span class="n">ceph</span><span class="o">.</span><span class="n">com</span><span class="o">/&lt;</span><span class="n">job</span><span class="o">-</span><span class="n">name</span><span class="o">&gt;/&lt;</span><span class="n">job</span><span class="o">-</span><span class="nb">id</span><span class="o">&gt;/</span>
</pre></div>
</div>
<p>or ssh into the teuthology server to view the results of the integration test:</p>
<blockquote>
<div><div class="highlight-default notranslate"><div class="highlight"><pre><style type="text/css">
span.prompt1:before {
  content: "$ ";
}
</style><span class="prompt1">ssh<span class="w"> </span>&lt;username&gt;@teuthology.front.sepia.ceph.com</span>
</pre></div></div></div></blockquote>
<p>and access <a class="reference external" href="../tests-integration-testing-teuthology-workflow/#teuthology-archives">teuthology archives</a>, as in this example:</p>
<blockquote>
<div><div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">nano<span class="w"> </span>/a/teuthology-2021-01-06_07:01:02-rados-master-distro-basic-smithi/</span>
</pre></div></div></div></blockquote>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>This requires you to have access to the Sepia lab. To learn how to
request access to the Sepia lab, see:
<a class="reference external" href="https://ceph.github.io/sepia/adding_users/">https://ceph.github.io/sepia/adding_users/</a></p>
</div>
</section>
<section id="identifying-failed-jobs">
<h2>Identifying Failed Jobs<a class="headerlink" href="#identifying-failed-jobs" title="Permalink to this heading"></a></h2>
<p>On pulpito, a job in red means either a failed job or a dead job. A job is
combination of daemons and configurations defined in the yaml fragments in
<a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa/suites">qa/suites</a> . Teuthology uses these configurations and runs the tasks listed
in <a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa/tasks">qa/tasks</a>, which are commands that set up the test environment and test
Ceph’s components. These tasks cover a large subset of use cases and help to
expose bugs not exposed by <a class="reference external" href="../tests-integration-testing-teuthology-intro/#make-check">make check</a> testing.</p>
<p>A job failure might be caused by one or more of the following reasons:</p>
<ul>
<li><p>environment setup (<a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa/distros/supported">testing on varied
systems</a>):
testing compatibility with stable realeases for supported versions.</p></li>
<li><p>permutation of config values: for instance, <a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa/suites/rados/thrash">qa/suites/rados/thrash</a> ensures
that we run thrashing tests against Ceph under stressful workloads so that we
can catch corner-case bugs. The final setup config yaml file used for testing
can be accessed at:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">/</span><span class="n">a</span><span class="o">/&lt;</span><span class="n">job</span><span class="o">-</span><span class="n">name</span><span class="o">&gt;/&lt;</span><span class="n">job</span><span class="o">-</span><span class="nb">id</span><span class="o">&gt;/</span><span class="n">orig</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">yaml</span>
</pre></div>
</div>
</li>
</ul>
<p>More details about config.yaml can be found at <a class="reference external" href="https://docs.ceph.com/projects/teuthology/en/latest/detailed_test_config.html">detailed test config</a></p>
</section>
<section id="triaging-the-cause-of-failure">
<h2>Triaging the cause of failure<a class="headerlink" href="#triaging-the-cause-of-failure" title="Permalink to this heading"></a></h2>
<p>When a job fails, you will need to read its teuthology log in order to triage
the cause of its failure. Use the job’s name and id from pulpito to locate your
failed job’s teuthology log:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">qa</span><span class="o">-</span><span class="n">proxy</span><span class="o">.</span><span class="n">ceph</span><span class="o">.</span><span class="n">com</span><span class="o">/&lt;</span><span class="n">job</span><span class="o">-</span><span class="n">name</span><span class="o">&gt;/&lt;</span><span class="n">job</span><span class="o">-</span><span class="nb">id</span><span class="o">&gt;/</span><span class="n">teuthology</span><span class="o">.</span><span class="n">log</span>
</pre></div>
</div>
<p>Open the log file:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">/</span><span class="n">a</span><span class="o">/&lt;</span><span class="n">job</span><span class="o">-</span><span class="n">name</span><span class="o">&gt;/&lt;</span><span class="n">job</span><span class="o">-</span><span class="nb">id</span><span class="o">&gt;/</span><span class="n">teuthology</span><span class="o">.</span><span class="n">log</span>
</pre></div>
</div>
<p>For example:</p>
<blockquote>
<div><div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">nano<span class="w"> </span>/a/teuthology-2021-01-06_07:01:02-rados-master-distro-basic-smithi/5759282/teuthology.log</span>
</pre></div></div></div></blockquote>
<p>Every job failure is recorded in the teuthology log as a Traceback and is
added to the job summary.</p>
<p>Find the <code class="docutils literal notranslate"><span class="pre">Traceback</span></code> keyword and search the call stack and the logs for
issues that caused the failure. Usually the traceback will include the command
that failed.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>The teuthology logs are deleted from time to time. If you are unable
to access the link in this example, just use any other case from
<a class="reference external" href="http://pulpito.front.sepia.ceph.com/">http://pulpito.front.sepia.ceph.com/</a></p>
</div>
</section>
<section id="reporting-the-issue">
<h2>Reporting the Issue<a class="headerlink" href="#reporting-the-issue" title="Permalink to this heading"></a></h2>
<p>In short: first check to see if your job failure was caused by a known issue,
and if it wasn’t, raise a tracker ticket.</p>
<p>After you have triaged the cause of the failure and you have determined that it
wasn’t caused by the changes that you made to the code, this might indicate
that you have encountered a known failure in the upstream branch (in the
example we’re considering in this section, the upstream branch is “octopus”).
If the failure was not caused by the changes you made to the code, go to
<a class="reference external" href="https://tracker.ceph.com">https://tracker.ceph.com</a> and look for tracker issues related to the failure by
using keywords spotted in the failure under investigation.</p>
<p>If you find a similar issue on <a class="reference external" href="https://tracker.ceph.com">https://tracker.ceph.com</a>, leave a comment on
that issue explaining the failure as you understand it and make sure to
include a link to your recent test run. If you don’t find a similar issue,
create a new tracker ticket for this issue and explain the cause of your job’s
failure as thoroughly as you can. If you’re not sure what caused the job’s
failure, ask one of the team members for help.</p>
</section>
<section id="debugging-an-issue-using-interactive-on-error">
<h2>Debugging an issue using interactive-on-error<a class="headerlink" href="#debugging-an-issue-using-interactive-on-error" title="Permalink to this heading"></a></h2>
<p>When you encounter a job failure during testing, you should attempt to
reproduce it. This is where <code class="docutils literal notranslate"><span class="pre">--interactive-on-error</span></code> comes in. This
section explains how to use <code class="docutils literal notranslate"><span class="pre">interactive-on-error</span></code> and what it does.</p>
<p>When you have verified that a job has failed, run the same job again in
teuthology but add the <a class="reference external" href="https://docs.ceph.com/projects/teuthology/en/latest/detailed_test_config.html#troubleshooting">interactive-on-error</a> flag:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>ideepika@teuthology:~/teuthology$ ./virtualenv/bin/teuthology -v --lock --block $&lt;your-config-yaml&gt; --interactive-on-error
</pre></div>
</div>
<p>Use either <a class="reference external" href="https://docs.ceph.com/projects/teuthology/en/latest/detailed_test_config.html#test-configuration">custom config.yaml</a> or the yaml file from the failed job. If
you use the yaml file from the failed job, copy <code class="docutils literal notranslate"><span class="pre">orig.config.yaml</span></code> to
your local directory:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>ideepika@teuthology:~/teuthology$ cp /a/teuthology-2021-01-06_07:01:02-rados-master-distro-basic-smithi/5759282/orig.config.yaml test.yaml
ideepika@teuthology:~/teuthology$ ./virtualenv/bin/teuthology -v --lock --block test.yaml --interactive-on-error
</pre></div>
</div>
<p>If a job fails when the <code class="docutils literal notranslate"><span class="pre">interactive-on-error</span></code> flag is used, teuthology
will lock the machines required by <code class="docutils literal notranslate"><span class="pre">config.yaml</span></code>. Teuthology will halt
the testing machines and hold them in the state that they were in at the
time of the job failure. You will be put into an interactive python
session. From there, you can ssh into the system to investigate the cause
of the job failure.</p>
<p>After you have investigated the failure, just terminate the session.
Teuthology will then clean up the session and unlock the machines.</p>
</section>
<section id="suggested-resources">
<h2>Suggested Resources<a class="headerlink" href="#suggested-resources" title="Permalink to this heading"></a></h2>
<blockquote>
<div><ul class="simple">
<li><p><a class="reference external" href="https://www.youtube.com/watch?v=gj1OXrKdSrs">Testing Ceph: Pains &amp; Pleasures</a></p></li>
</ul>
</div></blockquote>
</section>
</section>



<div id="support-the-ceph-foundation" class="admonition note">
  <p class="first admonition-title">Brought to you by the Ceph Foundation</p>
  <p class="last">The Ceph Documentation is a community resource funded and hosted by the non-profit <a href="https://ceph.io/en/foundation/">Ceph Foundation</a>. If you would like to support this and our other efforts, please consider <a href="https://ceph.io/en/foundation/join/">joining now</a>.</p>
</div>


           </div>
           
          </div>
          <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
        <a href="../tests-integration-testing-teuthology-workflow/" class="btn btn-neutral float-left" title="Integration Tests using Teuthology Workflow" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
        <a href="../tests-integration-testing-teuthology-kernel/" class="btn btn-neutral float-right" title="Integration Tests for Kernel Development" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>&#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).</p>
  </div>

   

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>